Discriminative Learning of Visual Data for Audiovisual Speech Recognition

In recent years a number of techniques have been proposed to improve the accuracy and the robustness of automatic speech recognitionin noisy environments. Among these, supplementing the acoustic information with visual data, mostly extracted from speaker's lip shapes, has been proved to be successful. We have already demonstrated the eeective-ness of integrating visual data at two diierent levels during speech decoding according to both direct and separate identiication strategies (DI+SI). This paper outlines methods for reinforcing the visible speech recognition in the framework of separate identiication. First, we deene visual-speciic units using a self-organizing mapping technique. Second, we complete a stochastic learning of these units with a discriminative neural-network-based technique for speech recognition purposes. Finally, we show on a connected-letter speech recognition task that using these methods improves performances of the DI+SI based system under varying noise-level conditions.